extend industry/subsectors to explicitly track FE by SE origin #1620

0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q · 2024-03-15T15:38:47Z

This extends industry/subsectors to explicitly track the SE-origin on FE demand. So far, we assumed identical SE-origin shares in FE demand across industry subsectors in order to report them (REMIND itself did not much care), but this collided with the assumptions made about chemical feedstocks (see #1561, #1576).

Before, we equated total industry FE supply (summed over SE origins) with industry FE demand (summed over subsectors). Now we introduce a new variable, v37_demFeIndst(t,regi,entySe,entyFe,in,emiMkt), for industry FE demand that is explicit in FE carrier, SE origin, and industry subsector (and emissions market). This allows for o37_demFeIndSub to report SE-specific subsector FE demand, without assuming SE-shares in FE carriers.

Two calibrations with these changes (SSP2EU-NPi and SSP2EU-EU21-NPi) are running under /p/tmp/pehl/Remind/ (_fast variants on the priority queue, others on medium).

New feature

Checklist:

My code follows the coding etiquette
I performed a self-review of my own code
I explained my changes within the PR, particularly in hard-to-understand areas
All automated model tests pass (FAIL 0 in the output of make test)
The changelog CHANGELOG.md has been updated correctly

mellamoSimon · 2024-03-15T15:49:06Z

modules/37_industry/subsectors/equations.gms

-      vm_demFENonEnergySector(t,regi,entySe2,entyFe,"indst",emiMkt)
-    )
-  =e=
+q37_feedstocksLimit(t,regi,entySe,entyFe,in,secInd37,emiMkt)$(


do we need this restriction with the new formulation of q37_demFeFeedstockChemIndst?

Which one? The rats-nest of lines 244-252? Actually, yes. This can be cleaned up by introducing some specialised mappings for the purpose, but the beauty treatment is scheduled for next week ;)

mellamoSimon · 2024-03-15T15:49:51Z

awesome! thank you for this, I think it will make things more tractable

0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q · 2024-03-15T16:08:57Z

@JakobBD PBS is widely infeasible with this (/p/tmp/pehl/Remind/output/SSP2EU_PBS-NPi-calibrate_fast_2024-03-15_16.45.24). Sorry about that, but need to wrap my head around those equations first and I am prioritising the "default" runs for now. I will look into that next week.

JakobBD · 2024-03-15T16:17:28Z

@JakobBD PBS is widely infeasible with this (/p/tmp/pehl/Remind/output/SSP2EU_PBS-NPi-calibrate_fast_2024-03-15_16.45.24). Sorry about that, but need to wrap my head around those equations first and I am prioritising the "default" runs for now. I will look into that next week.

What does that mean for the merging process? As it's a new feature, it may not have to be merged to develop right away? Do you think you could make the PR a WIP, do your runs in your own branch and wait with merging until the pbs part is sorted out?

0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q · 2024-03-15T16:22:43Z

Well, it is a new feature, but one that (hopefully) solves two problems – feedstock composition and solver performance.
I would not merge before the test runs come back good (or better then the status quo) and the PBS issue is sorted. I cannot say how desperate @LaviniaBaumstark is, though.

JakobBD · 2024-03-15T16:23:02Z

modules/37_industry/subsectors/equations.gms

+                                      AND secInd37_2_pf(secInd37,in)
+                                      AND secInd37_emiMkt(secInd37,emiMkt)
+                                      AND entyFe2Sector(entyFe,"indst")
+                                      AND entyFeCC37(entyFe)               ) ..


The entyFeCC37 one might be redundant. I did not want so spend a test run on finding out.

So is ppfen_industry_dyn37(in), made redundant by in_chemicals_feedstock_37(in), I suppose. But I wasn't even referring that, just looking in awe at 9 conditions...

JakobBD · 2024-03-15T16:41:40Z

modules/37_industry/subsectors/equations.gms

+                                      AND secInd37_emiMkt(secInd37,emiMkt) ) ..
+  sum((sefe(entySe,entyFe),
+       fe2ppfEn(entyFe,in)),
+    v37_demFeIndst(t,regi,entySe,entyFe,in,emiMkt)


That's actually quite a fundamental problem: ppfen do not exist any more for process-based sectors, so the whole variable definition (and thus the FE balance) is invalid for them.
Could you maybe define v37_demFeIndst(t,regi,entySe,entyFe,secInd37,emiMkt) instead? That would still give you the full information, except for the information of primary vs secondary steel. Otherwise, you'd have to skip that intermediate step for pbs and add its share directly to q37_demFeIndst - would be a shame to not have that tracking for pbs, and I don't know what implications that would have for reporting.
Sorry about that.

Bottom line: PR needs significant changes in order to not break the process-based implementation. @LaviniaBaumstark

JakobBD · 2024-03-15T16:56:40Z

Nice stuff otherwise! Sorry about the pbs hassle!

LaviniaBaumstark · 2024-03-16T11:03:43Z

many thanks for trying to improve teh situation! I agree that this is not ready for merging. Let's try to solve it on Monday and I will start calibration runs afterwards and we see how far we get for the validation. Maybe we will try to find a new date for a validation meeting before easter.

LaviniaBaumstark · 2024-03-16T11:07:14Z

Michaja's test runs seem to converge

0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q · 2024-03-19T09:09:20Z

Test runs here

/p/tmp/pehl/Remind_twin/output/SSP2EU-NPi-calibrate_2024-03-19_10.21.34/
/p/tmp/pehl/Remind_twin/output/SSP2EU-EU21-NPi-calibrate_2024-03-19_10.21.54/
/p/tmp/pehl/Remind_twin/output/SSP2EU_PBS-NPi-calibrate_2024-03-19_10.22.11/

0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q · 2024-03-19T09:09:51Z

Cleaning of conditions and sets fill be a follow-up.

JakobBD · 2024-03-19T10:58:02Z

modules/37_industry/subsectors/sets.gms

+  fe2mat(all_enty,all_enty,all_te)             ""   / /
+  secInd37_tePrc(secInd37,tePrc)               ""   / /
+  fe2ppfen_no_ces_use(all_enty,all_in)         ""   / /
+$endif.cm_subsec_model_steel


What is that needed for? Seems hard to maintain (whoever adds a set needs to know/remember there's this list lurking down here that it needs to be added to as well...)

Well, GAMS will help with a friendly compilation error in case something needs to be added here. :)
Only the sets used in set calculations down below, or somewhere else in the code, are required to be here. I just did not want to figure out one-by-one which those are.

Ah, you got rid of all the conditions above! Please revert that! Two reasons:

The one on maintainability I posted above

There will be other process-based subsectors soon. The sets are for all subsectors, the entries are subsector-specific. Your implementation will not allow switching implementations for sectors individually.

JakobBD · 2024-03-19T10:59:38Z

modules/37_industry/subsectors/bounds.gms

+*** be removed once the variable is established.
+loop ((t,regi,
+       sefe(entySe,entyFe),
+       fe2ppfEn(entyFe,in),


may need adaptation to ue?

In theory yes, but testOneRegi was not screaming bloody murder, so probably it is irrelevant now.

JakobBD · 2024-03-19T11:00:42Z

Looks good, thanks for the work, let's see what the calibration says.

close remindmodel#1613

- limit the changes in prices for selected production factors to between 0.5-2 of the price of the previous calibration iteration - this might stabilise calibration behaviour against sudden price jumps

…ead of ppfen to be compatible wit process-based implemen-based implementation

change

0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q · 2024-03-22T08:49:45Z

Status update

CES runs:
- speed ✔️: SSP2EU-NPi and SSP2EU-EU21-NPi calibrations look pretty fast, having completed eight and six calibration iterations in 17 hours. They might actually finish on the priority queue.
- buildings ✖️: fesob in EUR did not improve. CAZ seems to have the same problem, so more price debugging is necessary.
- industry ❓: looks really bad in EUR, JPN, NEU, USA, but is still oscillating, so with more calibration iterations it might settle. Unfortunately I did start these runs from stock gdxes, not ones from previous calibration attempts.
PBS run ✖️:
- SSP2EU_PBS-NPi on the other hand has not finished the first calibration iteration within 17 hours. It did 54 Nash iterations (SSP2EU-NPi did 26 in two and a half hours in the first calibration iteration), so this looks like a Nash convergence problem.
  
  That might go away if we use a gdx from the other calibration, or it might not.

I would merge this now, and tinker with calibration and Nash performance based on the develop branch again. @LaviniaBaumstark?

LaviniaBaumstark · 2024-03-22T08:58:13Z

I agree with merging and further working on improving the develop branch. Could you please also increase the input data version to 6.74

0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q · 2024-03-22T09:11:19Z

Could you please also increase the input data version to 6.74

I cannot.

remind/config/default.cfg

Line 30 in 54c883c

cfg$inputRevision <- "6.74"

LaviniaBaumstark · 2024-03-22T09:31:28Z

eeh, 6.75, but will do it later

Renato-Rodrigues · 2024-03-22T11:21:00Z

It does not solve the issue of slow convergence, but on the PBS run, maybe you could try to run the calibration relaxing the tax convergence criteria at first.
As it is a new structure, and we just recently enforced the tax criteria, maybe at initial calibration stages for a new structure we are asking too much from our framework.
Once an okish gdx is achieved, we could rerun the calibration with the tax criteria reenabled.

Renato-Rodrigues · 2024-03-22T11:24:18Z

PS: it would be nice to the person that is developing the PBS version to check also the gdx from the iteration that got infeasible running some testoneregi runs.
This iteration could give insight on which part of the model/region is actually difficult to solve on the formulation, and maybe point to the issue of slow convergence.

0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q · 2024-03-22T12:33:26Z

PS: it would be nice to the person that is developing the PBS version to check also the gdx from the iteration that got infeasible running some testoneregi runs. This iteration could give insight on which part of the model/region is actually difficult to solve on the formulation, and maybe point to the issue of slow convergence.

I do not think there is much information to be gained. In all likelihood, CONOPT just ran into some NA.

0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q · 2024-03-22T12:59:31Z

As expected, cannot replicate.

…5mncnisTJJ6q/fix/ind_feedstock_shares" This reverts commit 75039ce, reversing changes made to b6b0e58.

0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q requested review from LaviniaBaumstark, Renato-Rodrigues, JakobBD and mellamoSimon March 15, 2024 15:38

0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q force-pushed the fix/ind_feedstock_shares branch 2 times, most recently from 62a2115 to 120f5c7 Compare March 15, 2024 15:43

mellamoSimon reviewed Mar 15, 2024

View reviewed changes

mellamoSimon approved these changes Mar 15, 2024

View reviewed changes

JakobBD reviewed Mar 15, 2024

View reviewed changes

0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q force-pushed the fix/ind_feedstock_shares branch from fc6bfd6 to ab49a9a Compare March 19, 2024 08:59

0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q requested a review from JakobBD March 19, 2024 09:08

0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q force-pushed the fix/ind_feedstock_shares branch from ab49a9a to ca1752a Compare March 19, 2024 09:20

JakobBD reviewed Mar 19, 2024

View reviewed changes

0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q force-pushed the fix/ind_feedstock_shares branch 5 times, most recently from 2d8190f to 7a6a0ca Compare March 21, 2024 09:21

0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q and others added 6 commits March 21, 2024 15:43

proper sort-order for sety and fety

74334eb

reset to default CES parameterisation

0740868

extend industry/subsectors to explicitly track FE by SE origin

68df9e7

close remindmodel#1613

clean up after remindmodel#1614

9b5be02

limit price changes of pf druing calibration

11cf615

- limit the changes in prices for selected production factors to between 0.5-2 of the price of the previous calibration iteration - this might stabilise calibration behaviour against sudden price jumps

adapt q37_demFeIndst_intermediate: define v37_demFeIndst over ue inst…

8c21c66

…ead of ppfen to be compatible wit process-based implemen-based implementation

0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q force-pushed the fix/ind_feedstock_shares branch from 7a6a0ca to 8b8fb43 Compare March 21, 2024 14:43

0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q added 3 commits March 21, 2024 15:59

adapt q37_demFeIndst and o37_demFeIndSub to q37_demFeIndst_intermediate

29967df

change

use only vm_demFeSector_afterTax in industry module

1118863

lower minimum price for buildings ppf during calibration

25e9f6d

0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q force-pushed the fix/ind_feedstock_shares branch from 7b304ae to 25e9f6d Compare March 21, 2024 15:00

LaviniaBaumstark approved these changes Mar 22, 2024

View reviewed changes

0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q merged commit 75039ce into remindmodel:develop Mar 22, 2024
2 checks passed

mellamoSimon added the Chemicals Collection of PRs relevant for our new colleague modeling the chemical industry label Apr 3, 2024

JakobBD added a commit to JakobBD/remind that referenced this pull request Apr 8, 2024

Revert "Merge pull request remindmodel#1620 from 0UmfHxcvx5J7JoaOhFSs…

0ae12fe

…5mncnisTJJ6q/fix/ind_feedstock_shares" This reverts commit 75039ce, reversing changes made to b6b0e58.

JakobBD mentioned this pull request Apr 21, 2024

Selectively revert 1620: Remove SE from dimension from industry for performance #1659

Merged

14 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

extend industry/subsectors to explicitly track FE by SE origin #1620

extend industry/subsectors to explicitly track FE by SE origin #1620

0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q commented Mar 15, 2024

mellamoSimon Mar 15, 2024

0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q Mar 15, 2024

mellamoSimon commented Mar 15, 2024

0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q commented Mar 15, 2024

JakobBD commented Mar 15, 2024

0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q commented Mar 15, 2024

JakobBD Mar 15, 2024

0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q Mar 15, 2024

JakobBD Mar 15, 2024 •

edited

Loading

JakobBD Mar 15, 2024 •

edited

Loading

JakobBD Mar 15, 2024

JakobBD commented Mar 15, 2024

LaviniaBaumstark commented Mar 16, 2024

LaviniaBaumstark commented Mar 16, 2024

0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q commented Mar 19, 2024 •

edited

Loading

0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q commented Mar 19, 2024

JakobBD Mar 19, 2024

0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q Mar 19, 2024

JakobBD Mar 19, 2024 •

edited

Loading

JakobBD Mar 19, 2024

0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q Mar 19, 2024

JakobBD commented Mar 19, 2024

0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q commented Mar 22, 2024

LaviniaBaumstark commented Mar 22, 2024

0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q commented Mar 22, 2024

LaviniaBaumstark commented Mar 22, 2024

Renato-Rodrigues commented Mar 22, 2024

Renato-Rodrigues commented Mar 22, 2024

0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q commented Mar 22, 2024

0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q commented Mar 22, 2024

extend industry/subsectors to explicitly track FE by SE origin #1620

extend industry/subsectors to explicitly track FE by SE origin #1620

Conversation

0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q commented Mar 15, 2024

Checklist:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mellamoSimon commented Mar 15, 2024

0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q commented Mar 15, 2024

JakobBD commented Mar 15, 2024

0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q commented Mar 15, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JakobBD Mar 15, 2024 • edited Loading

Choose a reason for hiding this comment

JakobBD Mar 15, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JakobBD commented Mar 15, 2024

LaviniaBaumstark commented Mar 16, 2024

LaviniaBaumstark commented Mar 16, 2024

0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q commented Mar 19, 2024 • edited Loading

0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q commented Mar 19, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JakobBD Mar 19, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JakobBD commented Mar 19, 2024

0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q commented Mar 22, 2024

LaviniaBaumstark commented Mar 22, 2024

0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q commented Mar 22, 2024

LaviniaBaumstark commented Mar 22, 2024

Renato-Rodrigues commented Mar 22, 2024

Renato-Rodrigues commented Mar 22, 2024

0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q commented Mar 22, 2024

0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q commented Mar 22, 2024

JakobBD Mar 15, 2024 •

edited

Loading

JakobBD Mar 15, 2024 •

edited

Loading

0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q commented Mar 19, 2024 •

edited

Loading

JakobBD Mar 19, 2024 •

edited

Loading